Prioritized Experience Replay Research Articles

A deep reinforcement learning model for dynamic job-shop scheduling problem with uncertain processing time

The dynamic job-shop scheduling problem (DJSP) is a type of scheduling tasks where rescheduling is performed when encountering the uncertainties such as the uncertain operation processing time. However, the current deep reinforcement learning (DRL) scheduling approaches are hard to train convergent scheduling policies as the problem scale increases, which is very important for rescheduling under uncertainty. In this paper, we propose a DRL scheduling method for DJSP based on the proximal policy optimization (PPO) with hybrid prioritized experience replay. The job shop scheduling problem is formulated as a sequential decision-making problem based on Markov Decision Process (MDP) where a novel state representation is designed based on the feasible solution matrix which depicts the scheduling order of a scheduling task, a set of paired priority dispatching rules (PDR) are used as the actions and a new intuitive reward function is established based on the machine idle time. Moreover, a new hybrid prioritized experience replay method for PPO is proposed to reduce the training time where samples with positive temporal-difference (TD) error are replayed. Static experiments on classic benchmark instances show that the make-span obtained by our scheduling agent has been reduced by 1.59% on average than the best known DRL methods. In addition, dynamic experiments demonstrate that the training time of the reused scheduling policy is reduced by 27% compared with the retrained policy when encountering uncertainties such as uncertain operation processing time.

Exploring the Efficacy of Learning Techniques in Model Extraction Attacks on Image Classifiers: A Comparative Study

In the rapidly evolving landscape of cybersecurity, model extraction attacks pose a significant challenge, undermining the integrity of machine learning models by enabling adversaries to replicate proprietary algorithms without direct access. This paper presents a comprehensive study on model extraction attacks towards image classification models, focusing on the efficacy of various Deep Q-network (DQN) extensions for enhancing the performance of surrogate models. The goal is to identify the most efficient approaches for choosing images that optimize adversarial benefits. Additionally, we explore synthetic data generation techniques, including the Jacobian-based method, Linf-projected Gradient Descent (LinfPGD), and Fast Gradient Sign Method (FGSM) aiming to facilitate the training of adversary models with enhanced performance. Our investigation also extends to the realm of data-free model extraction attacks, examining their feasibility and performance under constrained query budgets. Our investigation extends to the comparison of these methods under constrained query budgets, where the Prioritized Experience Replay (PER) technique emerges as the most effective, outperforming other DQN extensions and synthetic data generation methods. Through rigorous experimentation, including multiple trials to ensure statistical significance, this work provides valuable insights into optimizing model extraction attacks.

VizNav: A Modular Off-Policy Deep Reinforcement Learning Framework for Vision-Based Autonomous UAV Navigation in 3D Dynamic Environments

Unmanned aerial vehicles (UAVs) provide benefits through eco-friendliness, cost-effectiveness, and reduction of human risk. Deep reinforcement learning (DRL) is widely used for autonomous UAV navigation; however, current techniques often oversimplify the environment or impose movement restrictions. Additionally, most vision-based systems lack precise depth perception, while range finders provide a limited environmental overview, and LiDAR is energy-intensive. To address these challenges, this paper proposes VizNav, a modular DRL-based framework for autonomous UAV navigation in dynamic 3D environments without imposing conventional mobility constraints. VizNav incorporates the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm with Prioritized Experience Replay and Importance Sampling (PER) to improve performance in continuous action spaces and mitigate overestimations. Additionally, VizNav employs depth map images (DMIs) to enhance visual navigation by accurately estimating objects’ depth information, thereby improving obstacle avoidance. Empirical results show that VizNav, by leveraging TD3, improves navigation, and the inclusion of PER and DMI further boosts performance. Furthermore, the deployment of VizNav across various experimental settings confirms its flexibility and adaptability. The framework’s architecture separates the agent’s learning from the training process, facilitating integration with various DRL algorithms, simulation environments, and reward functions. This modularity creates a potential to influence RL simulation in various autonomous navigation systems, including robotics control and autonomous vehicles.

Tugboat Scheduling Method Based on the NRPER-DDPG Algorithm: An Integrated DDPG Algorithm with Prioritized Experience Replay and Noise Reduction

Tugboat Scheduling Method Based on the NRPER-DDPG Algorithm: An Integrated DDPG Algorithm with Prioritized Experience Replay and Noise Reduction

Inspection Robot Navigation Based on Improved TD3 Algorithm.

The swift advancements in robotics have rendered navigation an essential task for mobile robots. While map-based navigation methods depend on global environmental maps for decision-making, their efficacy in unfamiliar or dynamic settings falls short. Current deep reinforcement learning navigation strategies can navigate successfully without pre-existing map data, yet they grapple with issues like inefficient training, slow convergence, and infrequent rewards. To tackle these challenges, this study introduces an improved two-delay depth deterministic policy gradient algorithm (LP-TD3) for local planning navigation. Initially, the integration of the long-short-term memory (LSTM) module with the Prioritized Experience Re-play (PER) mechanism into the existing TD3 framework was performed to optimize training and improve the efficiency of experience data utilization. Furthermore, the incorporation of an Intrinsic Curiosity Module (ICM) merges intrinsic with extrinsic rewards to tackle sparse reward problems and enhance exploratory behavior. Experimental evaluations using ROS and Gazebo simulators demonstrate that the proposed method outperforms the original on various performance metrics.

Advancements in Reinforcement Learning Algorithms for Autonomous Systems

Reinforcement learning, often known as RL, has developed as a strong paradigm to teach autonomous software agents to make choices in contexts that are both complicated and dynamic. This abstract investigates recent developments and uses of RL in a variety of fields, showing both its transformational potential and the constraints that it faces at present. Recent developments in reinforcement learning (RL) algorithms, in particular deep reinforcement learning (DRL), have made it possible to make major advancements in autonomous decision- making tasks. DRL algorithms can learn complicated representations of state-action spaces by using deep neural networks. This allows for more efficient exploration and exploitation methods to be implemented. Additionally, advancements in algorithmic enhancements, such as prioritized experience replay and distributional reinforcement learning, have improved the stability and sample efficiency of reinforcement learning algorithms, which has made it possible for these algorithms to be used in real-world applications. Robotics, autonomous cars, game playing, finance, and healthcare are just a few of the many fields that may benefit from the use of RL. In the field of robotics, reinforcement learning (RL) makes it possible for autonomous agents to learn how to navigate, manipulate, and move about in environments that are both complicated and unstructured. To improve both safety and efficiency on the road, autonomous cars make use of reinforcement learning (RL) to make decisions in dynamic traffic situations. In finance, RL algorithms are used for portfolio optimization, algorithmic trading, and risk management. These applications serve to improve investment techniques and decision-making procedures. Furthermore, in the field of healthcare, RL supports individualized treatment planning, clinical decision support, and medical image analysis, which enables physicians to provide patients with care that is specifically suited to their needs. Despite the promising improvements and applications, RL is still confronted with several difficulties that restrict its capacity to be widely adopted and scaled. Among these problems are the inefficiency of the sample, the trade-offs between exploration and exploitation, concerns about safety and dependability, and the need for explainability and interpretability in decision-making processes. To effectively address these difficulties, it is necessary to engage in collaborative efforts across several disciplines, conduct research on algorithmic developments, and establish extensive assessment frameworks (Anon, 2022).

Multi-agent collaborative path planning algorithm with reinforcement learning and combined prioritized experience replay in Internet of Things

As one of the most promising research areas in the field of reinforcement learning, multi-agent collaborative decision-making faces the challenge of real-time control in dynamic environments. This paper focuses on the path planning problem of multi-mobile sink collaborative data collection in the Internet of Things. Based on the excellent collaborative decision-making performance of multi-agent deep deterministic policy gradients (MADDPG), a novel path planning algorithm is proposed in the constructed wireless sensor network model. Furthermore, a combined prioritized experience replay strategy is designed to increase the utilization of important experiences in MADDPG. Extensive experiments conducted in various conditions show that compared with traditional MADDPG and MADDPG with prioritized experience replay, the proposed algorithm demonstrates the best performance. The accelerated convergence speed, enhanced training effectiveness and shortened paths indicate the capability of the proposed algorithm in multi-agent collaborative path planning problem.

Design and Experimental Validation of Deep Reinforcement Learning-Based Fast Trajectory Planning and Control for Mobile Robot in Unknown Environment.

This article is concerned with the problem of planning optimal maneuver trajectories and guiding the mobile robot toward target positions in uncertain environments for exploration purposes. A hierarchical deep learning-based control framework is proposed which consists of an upper level motion planning layer and a lower level waypoint tracking layer. In the motion planning phase, a recurrent deep neural network (RDNN)-based algorithm is adopted to predict the optimal maneuver profiles for the mobile robot. This approach is built upon a recently proposed idea of using deep neural networks (DNNs) to approximate the optimal motion trajectories, which has been validated that a fast approximation performance can be achieved. To further enhance the network prediction performance, a recurrent network model capable of fully exploiting the inherent relationship between preoptimized system state and control pairs is advocated. In the lower level, a deep reinforcement learning (DRL)-based collision-free control algorithm is established to achieve the waypoint tracking task in an uncertain environment (e.g., the existence of unexpected obstacles). Since this approach allows the control policy to directly learn from human demonstration data, the time required by the training process can be significantly reduced. Moreover, a noisy prioritized experience replay (PER) algorithm is proposed to improve the exploring rate of control policy. The effectiveness of applying the proposed deep learning-based control is validated by executing a number of simulation and experimental case studies. The simulation result shows that the proposed DRL method outperforms the vanilla PER algorithm in terms of training speed. Experimental videos are also uploaded, and the corresponding results confirm that the proposed strategy is able to fulfill the autonomous exploration mission with improved motion planning performance, enhanced collision avoidance ability, and less training time.

Deep reinforcement learning aided secure UAV communications in the presence of moving eavesdroppers

Integrating unmanned aerial vehicles(UAVs) with wireless systems is widely anticipated to bring enormous benefits to network users, and is considered as a key enabling technology for next generation wireless networks. However deployment of UAVs raises security concerns, as they are prone to eavesdropping attacks. In this paper we investigate secure data transmissions in a UAV-aided communication system from the physical layer security(PLS) perspective. The UAV is employed as a flying base station and transmit confidential information to a ground user. Specifically, an eavesdropper is moving in the vicinity of the ground user, attempting to intercept legitimate data transmission. We aim to optimize UAV trajectory to maximize secrecy throughput and due to the fast changing environment with unpredictable movements of the eave, this nonconvex problem is difficult to solve. Instead, we propose a deep reinforcement learning framework by reformulating this problem as a Markov Decision Process(MDP). By carefully designing state spaces,state-dependent action sets and time-dependant reward signals, we aim to help the agent learn an optimized trajectory and complete the flight task within time limit. We adapt the deep Q network(DQN) algorithm and extend it with prioritized experience replay(PER) and importance sampling method to ensure a stable and effective learning outcome. Simulation results validate the effectiveness of the learning algorithm and confirm that the agent is able to learn an effective policy with desirable and robust learning outcomes. Compared with benchmark schemes, our proposed scheme is shown to achieve considerable performance gain in secrecy throughput.

Decentralized computation offloading via multi-agent deep reinforcement learning for NOMA-assisted mobile edge computing with energy harvesting devices

Supporting latency-sensitive and computation-intensive applications is hardly possible for mobile devices (MDs) with limited battery capacity and low computing resources. Therefore, mobile edge computing (MEC) and wireless power transfer (WPT) have emerged as promising technologies that allow MDs to offload part or all of their workloads to MEC servers and harvest energy for prolonging their battery lifetime. However, the MEC server’s limited computing resources, available communication channel quality, and time-limited energy harvesting (EH) challenge the computation offloading. In this paper, we study the joint problem of decentralized computation offloading and resource allocation (JDCORA) in the environment of non-orthogonal multiple access (NOMA)-assisted MEC with multiple EH-enabled MDs. To learn decentralized offloading policies, we propose a multi-agent deep reinforcement learning (MADRL)-based scheme towards minimizing energy consumption and task completion time, which considers the cooperation between MDs to adjust their strategies. In particular, we improve the multi-agent deep deterministic policy gradient (MADDPG) by applying the features of double actors, double centralized critics, soft value estimation, critic regularization and proportional-based prioritized experience replay (pPER) and propose an algorithm called multi-agent twin actors regularized critics (MATARC). Simulation results demonstrate that the proposed MATARC has a better convergence performance compared to other baseline methods and also reduces the average energy consumption, task completion time and task drop rate.

Towards a fossil-free urban transport system: An intelligent cross-type transferable energy management framework based on deep transfer reinforcement learning

Deep reinforcement learning (DRL) is now a research focus for the energy management of fuel cell vehicles (FCVs) to improve hydrogen utilization efficiency. However, since DRL-based energy management strategies (EMSs) need to be retrained when the types of FCVs are changed, it is a laborious task to develop DRL-based EMSs for different FCVs. Given that, this article introduces transfer learning (TL) into DRL to design a novel deep transfer reinforcement learning (DTRL) method and then innovatively proposes an intelligent transferable energy management framework between two different urban FCVs based on the designed DTRL method to achieve the reuse of well-trained EMSs. To begin, an enhanced soft actor-critic (SAC) algorithm integrating prioritized experience replay (PER) is formulated to be the studied DRL algorithm in this article. Then, an enhanced-SAC based EMS of a light fuel cell hybrid electric vehicle (FCHEV) is pre-trained by using massive real-world driving data. After that, the learned knowledge stored in the FCHEV's well-trained EMS is captured and then transferred into the EMS of a heavy-duty fuel cell hybrid electric bus (FCHEB). Finally, the FCHEB's EMS is fine-tuned in a stochastic environment to ensure adaptability to real driving conditions. Simulation results indicate that, compared to the state-of-the-art baseline EMS, the proposed DTRL-based EMS accelerates the convergence speed by 91.55% and improves the fuel economy by 6.78%. This article contributes to shortening the development cycle of DRL-based EMSs and improving the utilization efficiency of hydrogen energy in the urban transport sector.

Actor Prioritized Experience Replay (Abstract Reprint)

A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical studies indicate that it considerably underperforms off-policy actor-critic algorithms. We theoretically show that actor networks cannot be effectively trained with transitions that have large TD errors. As a result, the approximate policy gradient computed under the Q-network diverges from the actual gradient computed under the optimal Q-function. Motivated by this, we introduce a novel experience replay sampling framework for actor-critic methods, which also regards issues with stability and recent findings behind the poor empirical performance of PER. The introduced algorithm suggests a new branch of improvements to PER and schedules effective and efficient training for both actor and critic networks. An extensive set of experiments verifies our theoretical findings, showing that our method outperforms competing approaches and achieves state-of-the-art results over the standard off-policy actor-critic algorithms.

Deep reinforcement learning based on balanced stratified prioritized experience replay for customer credit scoring in peer-to-peer lending

In recent years, deep reinforcement learning (DRL) models have been successfully utilised to solve various classification problems. However, these models have never been applied to customer credit scoring in peer-to-peer (P2P) lending. Moreover, the imbalanced class distribution in experience replay, which may affect the performance of DRL models, has rarely been considered. Therefore, this article proposes a novel DRL model, namely a deep Q-network based on a balanced stratified prioritized experience replay (DQN-BSPER) model, for customer credit scoring in P2P lending. Firstly, customer credit scoring is formulated as a discrete-time finite-Markov decision process. Subsequently, a balanced stratified prioritized experience replay technology is presented to optimize the loss function of the deep Q-network model. This technology can not only balance the numbers of minority and majority experience samples in the mini-batch by using stratified sampling technology but also select more important experience samples for replay based on the priority principle. To verify the model performance, four evaluation measures are introduced for the empirical analysis of two real-world customer credit scoring datasets in P2P lending. The experimental results show that the DQN-BSPER model can outperform four benchmark DRL models and seven traditional benchmark classification models. In addition, the DQN-BSPER model with a discount factor γ of 0.1 has excellent credit scoring performance.

Joint optimization of UAV communication connectivity and obstacle avoidance in urban environments using a double-map approach

AbstractCellular-connected unmanned aerial vehicles (UAVs), which have the potential to extend cellular services from the ground into the airspace, represent a promising technological advancement. However, the presence of communication coverage black holes among base stations and various obstacles within the aerial domain pose significant challenges to ensuring the safe operation of UAVs. This paper introduces a novel trajectory planning scheme, namely the double-map assisted UAV approach, which leverages deep reinforcement learning to address these challenges. The mission execution time, wireless connectivity, and obstacle avoidance are comprehensively modeled and analyzed in this approach, leading to the derivation of a novel joint optimization function. By utilizing an advanced technique known as dueling double deep Q network (D3QN), the objective function is optimized, while employing a mechanism of prioritized experience replay strengthens the training of effective samples. Furthermore, the connectivity and obstacle information collected by the UAV during flight are utilized to generate a map of radio and environmental data for simulating the flying process, thereby significantly reducing operational costs. The numerical results demonstrate that the proposed method effectively circumvents obstacles and areas with weak connections during flight, while also considering mission completion time.

Prioritized experience replay based on dynamics priority

Experience replay has been instrumental in achieving significant advancements in reinforcement learning by increasing the utilization of data. To further improve the sampling efficiency, prioritized experience replay (PER) was proposed. This algorithm prioritizes experiences based on the temporal difference error (TD error), enabling the agent to learn from more valuable experiences stored in the experience pool. While various prioritized algorithms have been proposed, they ignored the dynamic changes of experience value during the training process, merely combining different priority criteria in a fixed or linear manner. In this paper, we present a novel prioritized experience replay algorithm called PERDP, which employs a dynamic priority adjustment framework. PERDP adaptively adjusts the weights of each criterion based on average priority level of the experience pool and evaluates experiences’ value according to current network. We apply this algorithm to the SAC model and conduct experiments in the OpenAI Gym experimental environment. The experiment results demonstrate that the PERDP exhibits superior convergence speed when compared to the PER.

Dynamic path planning via Dueling Double Deep Q-Network (D3QN) with prioritized experience replay

Path planning is a key requirement for mobile robots employed for different tasks such as rescue or transport missions. Conventional methods such as A* or Dijkstra to tackle path planning problem need a premise map of the robot's environment. Nowadays, dynamic path planning is a popular research topic, which drives mobile robots without prior static requirements. Deep reinforcement learning (DRL), which is another popular research area, is being harnessed to solve dynamic path planning problem by the researchers. In this study, Deep Q-Networks, which is a subdomain of DRL are opted to solve dynamic path planning problem. We first employ well known techniques Double Deep Q-Networks (D2QN) and Dueling Double Deep Q-Networks (D3QN) to train a model which can drive a mobile robot in environments with static and dynamic obstacles within 3 different configurations. Then we propose D3QN with Prioritized Experience Replay (PER) extension in order to further optimize the DRL model. We created a test bed to measure the performance of the DRL models against 99 randomly generated goal locations. According to our experiments, D3QN-PER method performs better than D2QN and D3QN in terms of path length and travel time to the goal without any collisions. Robot Operating System and Gazebo simulation environment is utilized to realize the training and testing environments, thus, the trained DRL models can be deployed to any ROS compatible robot seamlessly.

Deep Reinforcement Learning-Based Hierarchical Energy Control Strategy of a Platoon of Connected Hybrid Electric Vehicles Through Cloud Platform

Due to the features of strong non-linearity and hybrid driving with multi-power sources, a novel deep reinforcement learning(DRL)-based hierarchical energy control architecture is presented for a group of connected hybrid electric vehicles (HEVs) in the platoon through cloud platform. Firstly, a higher-level model predictive control (MPC) law is designed to determine the desired acceleration of connected HEVs in a platoon. Secondly,a reward function with the change of state of charge (SOC) and instantaneous fuel consumption as independent variables is constructed, and the expert knowledge-based deep deterministic policy gradient (DDPG) method with prioritized experience replay (PER) is used for designing the lower-level energy management strategy of connected HEVs platooning. Then, to illustrate the rationality of decision-making by the presented DDPG strategy, the influence of state variables such as vehicle speed, desired acceleration and battery SOC on the agent action values of presented DDPG method is discussed. Finally, the test results show that the proposed control scheme can reasonably allocate the engine and motor powers in real time, and finally achieve the safe and energy-saving driving of connected HEVs in the platoon.

Deep reinforcement learning based active surge control for aeroengine compressors

This study proposes an active surge control method based on deep reinforcement learning to ensure the stability of compressors when adhering to the pressure rise command across the wide operating range of an aeroengine. Initially, the study establishes the compressor dynamic model with uncertainties, disturbances, and Close-Coupled Valve (CCV) actuator delay. Building upon this foundation, a Partially Observable Markov Decision Process (POMDP) is defined to facilitate active surge control. To address the issue of unobservability, a nonlinear state observer is designed using a finite-time high-order sliding mode. Furthermore, an Improved Soft Actor-Critic (ISAC) algorithm is developed, incorporating prioritized experience replay and adaptive temperature parameter techniques, to strike a balance between exploration and convergence during training. In addition, reasonable observation variables, error-segmented reward functions, and random initialization of model parameters are employed to enhance the robustness and generalization capability. Finally, to assess the effectiveness of the proposed method, numerical simulations are conducted, and it is compared with the fuzzy adaptive backstepping method and Second-Order Sliding Mode Control (SOSMC) method. The simulation results demonstrate that the deep reinforcement learning based controller outperforms other methods in both tracking accuracy and robustness. Consequently, the proposed active surge controller can effectively ensure stable operation of compressors in the high-pressure-ratio and high-efficiency region.

Memory-based soft actor–critic with prioritized experience replay for autonomous navigation

Memory-based soft actor–critic with prioritized experience replay for autonomous navigation

Single-lever control method design based on power management system and deep reinforcement learning for turboprop engines

This paper presents a single-lever control method based on Power Management System (PMS) and improved Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm for turboprop engines. In this approach, power level angle command, which is the single-lever command, is decoupled into controlled variable commands by PMS, and the controller based on improved TD3 algorithm can ensure that controlled variables track their commands rapidly and accurately. To achieve the optimal conversion relationship between different commands, an offline optimization process is used to design PMS. By optimization, specific fuel consumption and propeller efficiency are both improved after conversion. To deal with strong interactions between different control loops of a turboprop engine, TD3 algorithm which is a deep reinforcement learning algorithm is adopted. Two improvements which are the design method of observation state and prioritized experience replay are made to enhance the tracking accuracy. Simulation results show that improved TD3 algorithm can learn an optimal control policy to guarantee good control effect with fast response and small overshoot. The maximum settling time is less than 0.25s and the maximum overshoot is less than 0.1%. It also has a good robustness performance when the plant exists model uncertainties. The maximum fluctuations are less than 0.05%.